whittle down

verb

whittled down; whittling down; whittles down

: to gradually make (something) smaller by removing parts

We whittled the list down to four people.